Experimental migration support for POSIX #441
Conversation
AlCutter
force-pushed
the
posix_migrate
branch
6 times, most recently
from
61fc3f0 to
0980818
Compare
AlCutter
force-pushed
the
posix_migrate
branch
from
0980818 to
80e52e0
Compare
api/layout/paths.go
Outdated
| type RangeInfo struct { | ||
| // StartIndex is the index of the first entry bundle the range touches. | ||
| StartIndex uint64 | ||
| // StartPartial is non-zero if the bundle at StartIndex is expected to be partial. |
There was a problem hiding this comment.
Add something that this value references the first index into the bundle? i.e. (256 * StartIndex) + StartPartial is the index that this starts from?
At the moment the docs suggest this StartPartial is basically a boolean in disguise.
AlCutter
force-pushed
the
posix_migrate
branch
2 times, most recently
from
3bff6b5 to
44ee4f6
Compare
AlCutter
force-pushed
the
posix_migrate
branch
5 times, most recently
from
5f9abb3 to
efdee12
Compare
| todo chan bundle | ||
|
|
||
| // bundlesToMigrate is the total number of entry bundles which need to be copied. | ||
| bundlesToMigrate uint64 |
There was a problem hiding this comment.
Access to this field looks racy
There was a problem hiding this comment.
You're right - changed the approach
There was a problem hiding this comment.
It doesn't even need to be a field now. Can just be a local variable?
AlCutter
force-pushed
the
posix_migrate
branch
from
6d77318 to
c6143a5
Compare
Good call, but I'll add that in a follow up soon, this PR is already a bit large... |
| func Migrate(ctx context.Context, numWorkers int, sourceSize uint64, sourceRoot []byte, getEntries client.EntryBundleFetcherFunc, storage MigrationStorage) error { | ||
| klog.Infof("Starting migration; source size %d root %x", sourceSize, sourceRoot) | ||
|
|
||
| // TODO store state & resume |
There was a problem hiding this comment.
Naked TODO considered harmful. Before submit, can we have this fixed, a name attached, or an issue created and linked?
There was a problem hiding this comment.
I've removed the TODO - currently if the job crashes and is restarted, it'll pick up from the size of the locally integrated tree. That's likely good enough for now.
This PR adds experimental migration support for migrating into POSIX logs.
Towards #436